Fast Regression with an `∞ Guarantee∗

Sketching has emerged as a powerful technique for speeding up problems in numerical linear algebra, such as regression. In the overconstrained regression problem, one is given an n × d matrix A, with n d, as well as an n × 1 vector b, and one wants to find a vector x̂ so as to minimize the residual error ‖Ax− b‖2. Using the sketch and solve paradigm, one first computes S · A and S · b for a randomly chosen matrix S, then outputs x′ = (SA)†Sb so as to minimize ‖SAx′ − Sb‖2. The sketch-and-solve paradigm gives a bound on ‖x−x‖2 when A is well-conditioned. Our main result is that, when S is the subsampled randomized Fourier/Hadamard transform, the error x′ − x∗ behaves as if it lies in a “random” direction within this bound: for any fixed direction a ∈ R, we have with 1− d−c probability that 〈a, x′ − x∗〉 . ‖a‖2‖x ′ − x‖2